{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CPSC 330 Lecture 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lecture plan\n",
    "\n",
    "- 👋\n",
    "- **Turn on recording**\n",
    "- Announcements\n",
    "- True/False questions from last time (10 min)\n",
    "- Pipelines motivation (10 min)\n",
    "- Pipelines (15 min)\n",
    "- Break (5-10 min)\n",
    "- Hyperparameter optimization: grid search and random search (30 min)\n",
    "\n",
    "Bonus:\n",
    "\n",
    "- Bayesian hyperparameter optimization (10 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "- Apply scikit-learn `Pipeline`s to combine preprocessing with a classifier\n",
    "- Explain when/why using pipelines can prevent violations of the Golden Rule\n",
    "- Apply `GridSearchCV` and `RandomizedSearchCV` to tune hyperparameters of a pipeline\n",
    "- Explain why jointly optimizing hyperparameters is better than individually optimizing them\n",
    "- Compare the pros/cons of `GridSearchCV` vs. `RandomizedSearchCV`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['font.size'] = 16\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_validate, GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements\n",
    "\n",
    "- hw2 solutions posted.\n",
    "- hw3 posted, due Monday at 11:59pm.\n",
    "  - You can work with a partner, see instructions [here](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md#partners)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## True/False questions from last time (10 min)\n",
    "\n",
    "https://piazza.com/class/kb2e6nwu3uj23?cid=188"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipelines motivation (10 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to our dataset of the week, which is IMDB movie reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdb_df = pd.read_csv('data/imdb_master.csv', index_col=0, encoding=\"ISO-8859-1\")\n",
    "imdb_df = imdb_df[imdb_df['label'].str.startswith(('pos','neg'))]\n",
    "imdb_df = imdb_df.sample(frac=0.2, random_state=999) # Take a subsample of the dataset for speed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to split right away - better not even look at summary stats of the test data, or even eyeball it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdb_train, imdb_test = train_test_split(imdb_df, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a reminder, here is what we did last time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_imdb_raw = imdb_train['review']\n",
    "y_train_imdb = imdb_train['label']\n",
    "\n",
    "X_test_imdb_raw = imdb_test['review']\n",
    "y_test_imdb = imdb_test['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = CountVectorizer(min_df=50, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_imdb = vec.fit_transform(X_train_imdb_raw)\n",
    "X_test_imdb = vec.transform(X_test_imdb_raw);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "lr.fit(X_train_imdb, y_train_imdb);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9833333333333333"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_train_imdb, y_train_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8256"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_test_imdb, y_test_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last time, we avoided cross-validation. Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.82866667, 0.836     , 0.83733333, 0.83266667, 0.834     ])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(lr, X_train_imdb, y_train_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The code runs.\n",
    "- But we have a problem... our good friend the Golden Rule.\n",
    "- It is actually the exact same problem we fit/transformed the `CountVectorizer` before splitting.\n",
    "- Remember, cross-validation involves splitting!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_fold_1 = X_train_imdb[X_train_imdb.shape[0]//5:]\n",
    "X_valid_fold_1 = X_train_imdb[:X_train_imdb.shape[0]//5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But wait, the validation part was transformed using a `CountVectorizer` that was fit on the training split.\n",
    "- Just like last time, this is a Golden Rule violation.\n",
    "- For example, the validation split \"gets to be aware of\" words that are only in the training split.\n",
    "\n",
    "So what do we do here?\n",
    "\n",
    "![](img/hmm.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter pipelines to the rescue!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipelines (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- scikit-learn `Pipeline` can help us with this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This time we'll combine **the preprocessing and the model** with a `Pipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "countvec = CountVectorizer(min_df=50, binary=True)\n",
    "lr = LogisticRegression(max_iter=1000)\n",
    "\n",
    "pipe = Pipeline([\n",
    "    ('countvec', countvec),\n",
    "    ('lr', lr)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Syntax: pass in a list of **steps**.\n",
    "- The last step should be a model/classifier.\n",
    "- All the earlier steps should be transformers.\n",
    "  - Later in the course we'll see use cases for multiple rounds of transformers, here we only have one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.fit(X_train_imdb_raw, y_train_imdb);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What is this doing?\n",
    "- Note that I passed in the **raw** text data, not the vectorized word counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_imdb_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline is doing the following steps:\n",
    "\n",
    "1. Fitting `CountVectorizer`.\n",
    "2. Transforming the data using the fit `CountVectorizer`.\n",
    "3. Fitting the `LogisticRegression` on the transformed data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we call `predict` (or `score`), we also feed in the raw data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pos', 'pos', 'pos', ..., 'pos', 'pos', 'pos'], dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.predict(X_test_imdb_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a schematic assuming you have two transformers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/pipeline.png\" width=\"700\">\n",
    "\n",
    "[Source](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#18)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- One thing that is awesome here is that we can't make the mistakes we showed last time:\n",
    "  - We call `fit` on the train split and `score` on the test split, it's clean.\n",
    "  - We can't accidentally re-fit the preprocessor on the test data like we did last time.\n",
    "  - It automatically makes sure the same transformations are applied to train and test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, the moment of truth:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.82666667, 0.824     , 0.83133333, 0.83066667, 0.83533333])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(pipe, X_train_imdb_raw, y_train_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Remember what cross-validation does - it calls `fit` and `score`.\n",
    "- Now we're calling `fit` on the pipeline, not just the logistic regression.\n",
    "  - So **both the vectorizer and the logistic regression are refit again on each fold**.\n",
    "  - This is what we want to avoid the Golden Rule violation!\n",
    "  - Every validation score is unseen data with respect to the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/yay.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- BTW, the scores here aren't that different.\n",
    "- I don't suspect it matters all that much here.\n",
    "- But there could be cases where the effect is large.\n",
    "- In this course I want you to build good habits that will serve you well going forward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5-10 min)\n",
    "\n",
    "Please fill out the mid-course survey at https://ubc.ca1.qualtrics.com/jfe/form/SV_6tevNhMjZxRiQEl\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Hyperparameter optimization: grid search and random search (30 min)\n",
    "\n",
    "#### Manual hyperparameter optimization\n",
    "\n",
    "- We tried this a bit.\n",
    "- Advantage: we may have some intuition about what might work.\n",
    "  - E.g. if I'm massively overfitting, try decreasing `max_depth` or `C`.\n",
    "- Disadvantage: it takes a lot of work.\n",
    "- Disadvantage: in very complicated cases, our intuition might be worse than a data-driven approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Automated hyperparameter optimization\n",
    "\n",
    "- Advantage: reduce human effort\n",
    "- Advantage: less prone to error and improve reproducibility\n",
    "- Advantage: data-driven approaches may be effective\n",
    "- Disadvantage: may be hard to incorporate intuition\n",
    "- Disadvantage: be careful about overfitting on the validation set.\n",
    "\n",
    "\n",
    "\n",
    "There are two automated hyperparameter search methods in scikit-learn:\n",
    "\n",
    "  - Exhaustive grid search: [`sklearn.model_selection.GridSearchCV`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)\n",
    "  - Randomized hyperparameter optimization: [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "  \n",
    "The \"CV\" stands for cross-validation; these searchers have cross-validation built right in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Exhaustive grid search\n",
    "\n",
    "- A user specifies a set of values for each hyperparameter. \n",
    "- The method considers \"product\" of the sets and then evaluates each combination one by one.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start the automated hyperparameter optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    \"C\" : [0.01, 1, 10, 100]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression(max_iter=1000)\n",
    "grid_search = GridSearchCV(lr, param_grid, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that we can fix some hyperparameters and make others variable.\n",
    "- `verbose=1` tells `GridSearchCV` to print some output while it's working.\n",
    "  - This can be useful as this step sometimes takes a long time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 4 candidates, totalling 20 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:   11.8s finished\n"
     ]
    }
   ],
   "source": [
    "grid_search.fit(X_train_imdb, y_train_imdb);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going back to Lecture 3, this is what it's doing:\n",
    "\n",
    "```\n",
    "for C in [0.01, 1, 10, 100]:\n",
    "    for fold in folds:\n",
    "        fit in training portion with the given C\n",
    "        score on validation portion\n",
    "    compute average score\n",
    "pick hypers with best score\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here, we can extract the best hyperparameter values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 0.01}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8474666666666668"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the classifier inside like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.01, max_iter=1000)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pos', 'pos', 'pos', ..., 'pos', 'pos', 'pos'], dtype=object)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_.predict(X_test_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They also provide some \"syntactic sugar\" and allow you to call `predict` or `score` directly on the `GridSearchCV` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['pos', 'pos', 'pos', ..., 'pos', 'pos', 'pos'], dtype=object)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.predict(X_test_imdb) ## Does the same thing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the way, by default it takes the best hyperparameters and refits on the entire training set - very nice!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ?GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the `refit=True` to control this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ok, so this is all the syntax, but now we know we've been violating the Golden Rule because of the cross-validation.\n",
    "- Furthermore, we may want to tune the hyperparameters of the `CountVectorizer` and the `LogisticRegression` together.\n",
    "- Pipelines are perfect for this!!\n",
    "- So let's do it again properly this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "countvec = CountVectorizer(binary=True) # we should not set min_df here, it will be optimized\n",
    "lr = LogisticRegression(max_iter=1000)  # we should not set C here, it will be optimized\n",
    "\n",
    "pipe = Pipeline([\n",
    "    ('countvec', countvec),\n",
    "    ('lr', lr)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    \"countvec__min_df\" : [0, 10, 100],\n",
    "    \"lr__C\" : [0.01, 1, 10, 100]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Above: we have a nesting of transformers. \n",
    "- We can access the parameters of the \"inner\" objects by using `__` to go \"deeper\":\n",
    "  - `countvec__min_df`: \"the `min_df` of the `CountVectorizer` (of the pipeline)\"\n",
    "  - `lr__C`: \"the `C` of the `LogisticRegression` (of the pipeline)\"\n",
    "- Later in the course we'll see even deeper nesting, like `preprocessor__numeric__imputer__strategy`.\n",
    "\n",
    "So, now we pass int he `Pipeline` to `GridSearchCV`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(pipe, param_grid, verbose=2, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we pass in the raw text because we're using a `Pipeline`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:   23.7s\n",
      "[Parallel(n_jobs=-1)]: Done  60 out of  60 | elapsed:   44.0s finished\n"
     ]
    }
   ],
   "source": [
    "grid_search.fit(X_train_imdb_raw, y_train_imdb);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the `n_jobs=-1` above.\n",
    "- Hyperparameter optimization can be done _in parallel_ for each of the configurations.\n",
    "- This is very useful when scaling up to large numbers of machines in the cloud.\n",
    "- But even on my laptop there are 8 cores it can use, so that makes it a lot faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'countvec__min_df': 0, 'lr__C': 1}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Heh, here we get back the defaults again. This happens surprisingly often - the defaults are well chosen!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the number of candidates comes from the **product** of the number of options for each hyperparameter.\n",
    "- And then the whole thing multiplied by the number of folds (default is 5).\n",
    "- So, this number can get big really fast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But note that we're searching more possibilities than if we just sweep one hyperparameter at a time:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/gridsearch.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In that case we'd only get the ones in red, but here we get the entire grid.\n",
    "\n",
    "(Img source: see credit below)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Problems with exhaustive grid search \n",
    "\n",
    "- Required number of models to evaluate grows exponentially with the dimensionally of the configuration space. \n",
    "- Exhaustive search may become infeasible fairly quickly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Randomized hyperparameter search\n",
    "\n",
    "- Randomized hyperparameter optimization: [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "- Samples configurations at random until certain budget (e.g., time) is exhausted.\n",
    "- Advantage: you can choose how many runs you'll do.\n",
    "- Advantage: you can restrict yourself less on what values you might try.\n",
    "- Advantage: Adding parameters that do not influence the performance does not affect efficiency.\n",
    "- Advantage: research shows this is generally a better idea than grid search, see image for intuition:\n",
    "\n",
    "![](img/randomsearch_bergstra.png)\n",
    "\n",
    "Source: [Bergstra and Bengio, Random Search for Hyper-Parameter Optimization, JMLR 2012](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).\n",
    "\n",
    "- You don't know in advance which hyperparameters are important for your problem.\n",
    "- But some of them might be unimportant.\n",
    "- In the left figure, 6 of the 9 searches are useless because they are only varying the unimportant parameter.\n",
    "- In the right figure, all 9 searches are useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back to syntax. We can have continue to have the parameters chosen from a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_choices = {\n",
    "    \"countvec__min_df\" : [0, 10, 100],\n",
    "    \"lr__C\" : [0.01, 1, 10, 100]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In which case it will randomly pick from each list for each run. \n",
    "- So, we can afford to have many more values this time since we're not going to try all of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_choices = {\n",
    "    \"countvec__min_df\" : np.arange(0,100),\n",
    "    \"lr__C\" : 2.0**np.arange(-5,5)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.03125,  0.0625 ,  0.125  ,  0.25   ,  0.5    ,  1.     ,\n",
       "        2.     ,  4.     ,  8.     , 16.     ])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2.0**np.arange(-5,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the exponential range for `C`. This is quite common.\n",
    "- There is no point trying $C=\\{1,2,3\\ldots,100\\}$ because $C=1,2,3$ are too similar to each other.\n",
    "- Often we're trying to find an order of magnitude, e.g. $C=\\{0.01,0.1,1,10,100\\}$. \n",
    "- We can also write that is $C=\\{10^{-2},10^{-1},10^0,10^1,10^2\\}$. \n",
    "- Or, in other words, $C$ values to try are $10^n$ for $n=-2,-1,0,1,2$ which is basically what we have above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another thing we can do is give probability distributions to draw from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_choices = {\n",
    "    \"countvec__min_df\" : scipy.stats.randint(low=0, high=300),\n",
    "    \"lr__C\" : scipy.stats.randint(low=0, high=300) # TODO: this is lame, pick a continuous prob dist\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This is a bit fancy. What's nice is that you can have it concentrate more on certain values by setting the distribution. \n",
    "- Let's go back to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_choices = {\n",
    "    \"countvec__min_df\" : np.arange(0,100),\n",
    "    \"lr__C\" : 2.0**np.arange(-5,5)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_search = RandomizedSearchCV(pipe, param_choices,\n",
    "                                   n_iter = 12, \n",
    "                                   verbose = 1,\n",
    "                                   n_jobs = -1,\n",
    "                                   random_state = 123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note the `n_iter`, we didn't need this for `GridSearchCV`.\n",
    "  - Larger `n_iter` will take longer but do more searching.\n",
    "  - Remember you still need to multiply by number of folds!\n",
    "- I also set `random_state` but you don't have it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:   21.4s\n",
      "[Parallel(n_jobs=-1)]: Done  60 out of  60 | elapsed:   33.4s finished\n"
     ]
    }
   ],
   "source": [
    "random_search.fit(X_train_imdb_raw, y_train_imdb);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note: some hyperparameters significantly affect the training time!\n",
    "- For example, setting `min_df=0` is going to be much slower than `min_df=50`.\n",
    "  - Because the former results in way more columns.\n",
    "  - That makes everything slower, including the logistic regression training.\n",
    "  - You'll see this on hw3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'lr__C': 0.0625, 'countvec__min_df': 13}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now we get something different! \n",
    "- What's the score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8605333333333333"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8561333333333334"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- So, they are very slightly different.\n",
    "- Is that difference important?\n",
    "- Do we BELIEVE that difference? How to figure this out?\n",
    "\n",
    "<br><br>\n",
    "\n",
    "\n",
    "- Some strategies:\n",
    "  - We can try it out on the test set.\n",
    "  - We can look at the sub-scores of the folds.\n",
    "  - Try cross-validation with more folds.\n",
    "\n",
    "But first:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#??RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time', 'param_lr__C', 'param_countvec__min_df', 'params', 'split0_test_score', 'split1_test_score', 'split2_test_score', 'split3_test_score', 'split4_test_score', 'mean_test_score', 'std_test_score', 'rank_test_score'])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.cv_results_.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>param_countvec__min_df</th>\n",
       "      <th>param_lr__C</th>\n",
       "      <th>mean_fit_time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rank_test_score</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.860533</td>\n",
       "      <td>13</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>2.644730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.860267</td>\n",
       "      <td>5</td>\n",
       "      <td>0.03125</td>\n",
       "      <td>3.331615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.858533</td>\n",
       "      <td>22</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>2.490565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.853200</td>\n",
       "      <td>20</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.042938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.853067</td>\n",
       "      <td>46</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>3.052222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.849200</td>\n",
       "      <td>63</td>\n",
       "      <td>0.125</td>\n",
       "      <td>2.913377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.845600</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>3.478485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.840400</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>4.030078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.832000</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>3.735136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.829600</td>\n",
       "      <td>92</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.393095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.822800</td>\n",
       "      <td>32</td>\n",
       "      <td>8</td>\n",
       "      <td>3.785914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.810667</td>\n",
       "      <td>47</td>\n",
       "      <td>16</td>\n",
       "      <td>4.644994</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 mean_test_score param_countvec__min_df param_lr__C  \\\n",
       "rank_test_score                                                       \n",
       "1                       0.860533                     13      0.0625   \n",
       "2                       0.860267                      5     0.03125   \n",
       "3                       0.858533                     22      0.0625   \n",
       "4                       0.853200                     20        0.25   \n",
       "5                       0.853067                     46      0.0625   \n",
       "6                       0.849200                     63       0.125   \n",
       "7                       0.845600                     19           1   \n",
       "8                       0.840400                     13           8   \n",
       "9                       0.832000                     58           1   \n",
       "10                      0.829600                     92         0.5   \n",
       "11                      0.822800                     32           8   \n",
       "12                      0.810667                     47          16   \n",
       "\n",
       "                 mean_fit_time  \n",
       "rank_test_score                 \n",
       "1                     2.644730  \n",
       "2                     3.331615  \n",
       "3                     2.490565  \n",
       "4                     3.042938  \n",
       "5                     3.052222  \n",
       "6                     2.913377  \n",
       "7                     3.478485  \n",
       "8                     4.030078  \n",
       "9                     3.735136  \n",
       "10                    3.393095  \n",
       "11                    3.785914  \n",
       "12                    4.644994  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(random_search.cv_results_)[['mean_test_score', 'param_countvec__min_df', 'param_lr__C', 'mean_fit_time', 'rank_test_score']].set_index(\"rank_test_score\").sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Look at the timings, they can be quite interesting (try this on the homework!).\n",
    "- And now, the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8556"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8556"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8544"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "- `Pipeline`s combine any number of transformers with a model/estimator\n",
    "- Without a pipeline, one may violate the Golden Rule during cross-validation\n",
    "  - Data in a validation split has been preprocessed usng information from the training split\n",
    "- A pipeline's `.fit()` fits all the transformers and the classifier\n",
    "- A pipeline's `.predict()` calls `transform()` on all the transformers and then `predict` on the classifier.\n",
    "- `GridSearchCV` and `RandomizedSearchCV` are useful tools for automated  hyperparameter tuning\n",
    "- `GridSearchCV` explores all possibilities of the given hyperparameter space\n",
    "- `RandomizedSearchCV` gives more flexibility in terms of the number of iterations, and may be better in general.\n",
    "- With `RandomizedSearchCV` one does not need to specify a \"grid\" on numeric hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayesian hyperparameter optimization (bonus)\n",
    "\n",
    "- Both `GridSearchCV` and `RandomizedSearchCV` do each trial independently.\n",
    "- What if you could learn from your experience, e.g. learn that `max_depth=3` is bad?\n",
    "  - That could save time because you wouldn't try combinations involving `max_depth=3` in the future.\n",
    "- We can do this with `scikit-optimize`, which is a completely different package from `scikit-learn`\n",
    "- It uses a technique called \"model-based optimization\" and we'll specifically use \"Bayesian optimization\".\n",
    "  - In short, it uses machine learning to predict what hyperparameters will be good.\n",
    "  - Machine learning on machine learning!\n",
    "- As it happens I did my PhD thesis on this topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import BayesSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `BayesSearchCV` uses the same interface as `GridSearchCV` and `RandomSearchCV`.\n",
    "- However, the way we specify the parameter distributions is slightly different.\n",
    "- Here, we can just give the bounds as tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "bayes_opt = BayesSearchCV(\n",
    "    pipe,\n",
    "    {\n",
    "        'countvec__min_df': (0, 300),   # This gets interpreted as a range\n",
    "        'lr__C': (0.25, 0.5, 1, 2, 4, 8, 16, 32) # This gets interpreted as a list.\n",
    "    },\n",
    "    n_iter=10,\n",
    "    cv=3,\n",
    "    random_state=123,\n",
    "    verbose=0,\n",
    "    refit=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 1s, sys: 2.53 s, total: 1min 3s\n",
      "Wall time: 55.3 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BayesSearchCV(cv=3,\n",
       "              estimator=Pipeline(steps=[('countvec',\n",
       "                                         CountVectorizer(binary=True)),\n",
       "                                        ('lr',\n",
       "                                         LogisticRegression(max_iter=1000))]),\n",
       "              n_iter=10, random_state=123,\n",
       "              search_spaces={'countvec__min_df': (0, 300),\n",
       "                             'lr__C': (0.25, 0.5, 1, 2, 4, 8, 16, 32)})"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "bayes_opt.fit(X_train_imdb_raw, y_train_imdb);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It took longer than the other methods.\n",
    "  - In reality there is some extra computation to do the \"meta-ML\".\n",
    "  - This is not that significant here.\n",
    "  - The overall time is dominated by the time of calling `fit`.\n",
    "  - Another reason it look longer is because of the `n_jobs` issue (more on this below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('countvec__min_df', 8), ('lr__C', 32.0)])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_opt.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.844"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_opt.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In theory, it should get even better as we increase `n_iter` (because it has more data to learn from).\n",
    "- Checking the test score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.84"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_opt.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And reproducing the previous test scores for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8544"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8556"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.score(X_test_imdb_raw, y_test_imdb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In this case, this didn't seem very effective. Sometimes it is. Especially with more hyperparameters and more trials.\n",
    "- The more trials you do, the more it can learn from past experience.\n",
    "- Should I always use this? Not necessarily. (Heh, well I guess this is one case where it wasn't great.)\n",
    "- Disadvantage: requires installation.\n",
    "- Disadvantage: when number of trials is large (e.g. hundreds), the meta-ML can actually get too slow.\n",
    "- Disadvantage: harder parallelize the search because each trial depends on the previous ones.\n",
    "  - Note `n_jobs` parameter for `GridSearchCV` and `RandomizedSearchCV`.  \n",
    "  - `BayesSearchCV` also has this parameter.\n",
    "  - It can definitely parallelize the folds.\n",
    "  - The search will be less effective if it parallelizes further.\n",
    "- I feel there's kind of a \"sweet spot\" of maybe ~10 continuous hyperparameters and ~100 trials where this tends to do really well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Can I generalize this to say `BayesSearchCV` > `RandomizedSearchCV` > `GridSearchCV`?\n",
    "- Not quite. I'd say `RandomizedSearchCV` > `GridSearchCV` is pretty reasonable most of the time.\n",
    "- But we should think a bit more carefully about `BayesSearchCV` for the above reasons.\n",
    "- `RandomizedSearchCV` is often a reasonable choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A\n",
    "\n",
    "(Pause for Q&A)\n",
    "\n",
    "<br><br><br><br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
